Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SMTP server email notification example #386

Merged
merged 5 commits into from
Jan 11, 2021

Conversation

Tomcli
Copy link
Member

@Tomcli Tomcli commented Dec 7, 2020

Which issue is resolved by this Pull Request:
related #366

Description of your changes:
Add SMTP server email notification example based on the send-email Tekton catalog. We don't add this to the sample readme list yet because SMTP server required a paid account which is not open source friendly.

For other type of notifications, the Watson notification API should able to handle it with a simple HTTP/S call.

Environment tested:

  • Python Version (use python --version): 3.7
  • Tekton Version (use tkn version): 1.16
  • Kubernetes Version (use kubectl version): 1.18
  • OS (e.g. from /etc/os-release):

Checklist:

  • The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

    PR titles examples:

    • fix(frontend): fixes empty page. Fixes #1234
      Use fix to indicate that this PR fixes a bug.
    • feat(backend): configurable service account. Fixes #1234, fixes #1235
      Use feat to indicate that this PR adds a new feature.
    • chore: set up changelog generation tools
      Use chore to indicate that this PR makes some changes that users don't need to know.
    • test: fix CI failure. Part of #1234
      Use part of to indicate that a PR is working on an issue, but shouldn't close the issue when merged.
  • Do you want this pull request (PR) cherry-picked into the current release branch?

    If yes, use one of the following options:

    • (Recommended.) Ask the PR approver to add the cherrypick-approved label to this PR. The release manager adds this PR to the release branch in a batch update.
    • After this PR is merged, create a cherry-pick PR to add these changes to the release branch. (For more information about creating a cherry-pick PR, see the Kubeflow Pipelines release guide.)

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Tomcli
Copy link
Member Author

Tomcli commented Dec 7, 2020

/cc @fenglixa

@Tomcli
Copy link
Member Author

Tomcli commented Dec 7, 2020

You can extend the component script with the additional SMTP server requirements you need.

_parser.add_argument("--subject", type=str, required=True)
_parser.add_argument("--body", type=str, required=True)
_parser.add_argument("--sender", type=str, required=True)
_parser.add_argument("--recipients", type=str, required=True)
Copy link
Member

@fenglixa fenglixa Dec 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Tommy, could you please add attachment support?
Maybe refer to: https://stackoverflow.com/a/3363254

Copy link
Member Author

@Tomcli Tomcli Dec 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do you want to pass the attachment in kfp? using big data passing (via workspace) or volumeOP? By default, i can make use of big data passing and volumeOP to create a new pvc to pass the attachment for each pipeline.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

big data passing is much more meanful for this case, as the case of the attachment should be from client, but not server.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So right now here is the component.yaml for sending just one attachment file:

name: sendmail
description: |
  This task sends a simple email to receivers via SMTP server
inputs:
  - {name: server, description: 'secret name for SMTP server information (url, port, password)'}
  - {name: subject, description: 'plain text email subject'}
  - {name: body, description: 'plain text email body'}
  - {name: sender, description: 'sender email address'}
  - {name: recipients, description: 'recipient email addresses (space delimited list)'}
  - {name: attachment_path, description: 'email attachment file path'}
implementation:
  container:
    image: docker.io/library/python:3.8-alpine@sha256:c31682a549a3cc0a02f694a29aed07fd252ad05935a8560237aed99b8e87bf77 #tag: 3.8-alpine
    command:
    - python3
    - -u
    - -c
    - |
      #!/usr/bin/env python3
      import argparse
      _parser = argparse.ArgumentParser('sendmail inputs')
      _parser.add_argument("--server", type=str, required=True)
      _parser.add_argument("--subject", type=str, required=True)
      _parser.add_argument("--body", type=str, required=True)
      _parser.add_argument("--sender", type=str, required=True)
      _parser.add_argument("--recipients", type=str, required=True)
      _parser.add_argument("--attachment_path", type=str, default='')
      _parsed_args = _parser.parse_args()

      import smtplib, ssl, os
      from pathlib import Path
      from email.mime.multipart import MIMEMultipart
      from email.mime.base import MIMEBase
      from email.mime.text import MIMEText
      from email.utils import COMMASPACE, formatdate
      from email import encoders
      port = os.getenv('PORT')
      smtp_server = os.getenv('SERVER')
      sender_email = "$(params.sender)"
      receiver_emails = "$(params.recipients)"
      user = os.getenv('USER')
      password = os.getenv('PASSWORD')
      tls = os.getenv('TLS')
      path = _parsed_args.attachment_path

      msg = MIMEMultipart()
      msg['From'] = sender_email
      msg['To'] = receiver_emails
      msg['Subject'] = "$(params.subject)"
      msg.attach(MIMEText("$(params.body)"))
      if tls == 'True':
          context = ssl.create_default_context()
          server = smtplib.SMTP_SSL(smtp_server, port, context=context)
      else:
          server = smtplib.SMTP(smtp_server, port)
      if password != '':
          server.login(user, password)
      part = MIMEBase('application', "octet-stream")
      with open(path, 'rb') as file:
          part.set_payload(file.read())
      encoders.encode_base64(part)
      part.add_header('Content-Disposition',
                      'attachment; filename="{}"'.format(Path(path).name))
      msg.attach(part)
      for receiver in receiver_emails.split(' '):
          server.sendmail(sender_email, receiver, msg.as_string())
      server.quit()
    args: [
      --server, {inputValue: server},
      --subject, {inputValue: subject},
      --body, {inputValue: body},
      --sender, {inputValue: sender},
      --recipients, {inputValue: recipients},
      --attachment_path, {inputPath: attachment_path},
    ]

Copy link
Member Author

@Tomcli Tomcli Dec 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if I want to make the containerOp or component.yaml to take multiple attachments from multiple output paths, we will start seeing issue like #315 because kfp needs to have explicit arguments for their functions.

Copy link
Member Author

@Tomcli Tomcli Dec 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A workaround I could think of is wrapping the python code into a custom container that can take a list arguments. Then, for each new pipeline, the user can define a new ContainerOp using that custom container to change the number of attachments in the list. This is less friendly than component.yaml, but at least we don't have to change the DSL code to support this feature.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, what service do you use for sending emails through SMTP? I have a trial version from Sendgrid, but having some weird problem when sending emails with attachments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One attachment should be OK for the example.

Copy link
Member Author

@Tomcli Tomcli Dec 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so because we are running the notification task as exit handler, we cannot make sure any task will complete in the pipeline. Therefore, KFP doesn't allow dependency to be made for the exit task.

Since we cannot create a output path dependency for the exit handler task, we cannot use the big data passing to map the workspace value for us. So right now I just create a pvc and assume the regular task will complete and write the file to the assumed path. It also can be done with volumeOp, but volumeOp will create a new pvc for every pipeline.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes, you are right, the exit handler has limitition here to run a pipeline behind it.

@fenglixa
Copy link
Member

fenglixa commented Dec 8, 2020

Thanks @Tomcli! Just a minior request appended.

Copy link
Member

@ckadner ckadner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, just a small code formatting nit pick :-)

samples/exit-handler-email/send-email.py Outdated Show resolved Hide resolved
@fenglixa
Copy link
Member

fenglixa commented Jan 8, 2021

@Tomcli , is this PR ready to merge? I have no other comments except attachment support.

@Tomcli
Copy link
Member Author

Tomcli commented Jan 8, 2021

@fenglixa you can put lgtm if it looks good to you. we can enhance the file passing for finally once finally can access the task results. Hopefully in the up coming Tekton 0.20.
tektoncd/pipeline#3242

@fenglixa
Copy link
Member

OK, Thanks @Tomcli
/lgtm

@k8s-ci-robot k8s-ci-robot merged commit d8a9d5b into kubeflow:master Jan 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants